Skip to content

Add support for UV dependency management #2601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add support for UV dependency management #2601

wants to merge 3 commits into from

Conversation

@geruh
Copy link
Contributor

@geruh geruh commented Oct 10, 2025 •
edited
Loading

Closes #2553

Rationale for this change

During the release process cleanup, I also had prototyped a migration from Poetry to UV. I used the https://github.com/mkniewallner/migrate-to-uv package to migrate to UV and ran a release workflow and some others like doc serve for validation.

This PR also migrates the Makefile to use UV. I'll still need to upgrade the docs to reflect the UV contribution workflow. I think it would be a good start to push up as a prototype, and have others checkout and give it a try.

Build System Changes

Build Backend

UV doesn't have a build backend like poetry that supports hooks or custom build scripts. So I've switched to use Hatchling as the build backend with the existing (build-module.py) to ensure Avro files are properly compiled and included.

Build Verification

Release builds can be verified using:

uv build --sdist  # Build source distribution
uv build --wheel  # Build wheel distribution

Both commands ensure parity with the previous Poetry build outputs.

Are these changes tested?

Yes

  • make install Environment setup and dependency installation
  • make test - Unit tests
  • make test-integration - Integration tests with all extras
  • make lint - Linting and code quality checks
  • make doc-install && make doc-serve - Documentation

Are there any user-facing changes?

No user facing changes. Contributors will need to use UV instead of Poetry for development, but all make targets remain the same. So not really?

Fokko
Fokko approved these changes Oct 12, 2025
View reviewed changes
Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great @geruh, I hear a lot of good stuff around uv 🙌

pyproject.toml
"Programming Language :: Python :: 3.12",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3.9",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one, maybe we should add pretty-format-toml to pre-commit to format the toml files 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a great idea. #2605 if you want to merge it in

pyproject.toml

[project.optional-dependencies]
pyarrow = [
"pyarrow>=17.0.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, maybe find a way to deduplicate the constraints. pyarrow is mentioned multiple times.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! it looks like poetry had a way of doing this before with the [tool.poetry.extras], kind of acting as a version catalog. UV doesn't have support for this but respects the PEP standard for project dependencies. In this case one way we can de-duplicate is by using the PEP 508 Extras grammar which allows for self referencing other dependencies in the project like:

[project.optional-dependencies]
pyarrow = [
    "pyarrow>=17.0.0",
    "pyiceberg-core>=0.5.1,<0.7.0",
]

pandas = [
    "pyiceberg[pyarrow]",
    "pandas>=1.0.0,<3.0.0",
]

What do you think @Fokko?

@rambleraptor rambleraptor mentioned this pull request Oct 13, 2025
Open
kevinjqliu
kevinjqliu reviewed Oct 26, 2025
View reviewed changes
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

I tested a few things locally.

  • all the make command
  • side by side pyproject.toml comparison
  • ran a few github workflows on my forked repo
    • python-ci.yml ✅
    • python-ci-docs.yml ✅
    • nightly-pypi-build.yml ❌ (something weird with cibuildwheel)

We might want to rebase this PR to capture of of the recent changes, such as the 3.9 deprecation. I'm not too worried about the library versioning since the dependabot workflow will upgrade them again

btw, there are a still references to poetry in mkdocs/docs/contributing.md

pyproject.toml
authors = ["Apache Software Foundation <dev@iceberg.apache.org>"]
license = "Apache License 2.0"
authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
requires-python = ">=3.9.2,<4.0.0,!=3.9.7"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
requires-python = ">=3.9.2,<4.0.0,!=3.9.7"
requires-python = ">=3.10,<4.0.0"

we just removed 3.9 in #2554

pyproject.toml
"Programming Language :: Python :: 3.12",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3.9",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"Programming Language :: Python :: 3.9",

.github/workflows/svn-build-artifacts.yml
Comment on lines +50 to +51
- name: Install UV
run: make install-uv
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we need to use

      - name: Install uv
        uses: astral-sh/setup-uv@v6

instead of make install-uv

When i was testing "Nightly PyPI Build", make install-uv fails for windows machine with

Run uv version "0.10.0.dev20251026230001"
uv: D:\a\_temp\c958d63c-abd9-4fae-8896-0d96857f661f.ps1:2
Line |
   2 |  uv version "0.10.0.dev20251026230001"
     |  ~~
     | The term 'uv' is not recognized as a name of a cmdlet, function, script file, or executable program. Check the
     | spelling of the name, or if a path was included, verify that the path is correct and try again.
Error: Process completed with exit code 1.

https://github.com/kevinjqliu/iceberg-python/actions/runs/18824958012/job/53706130681

@kevinjqliu
Copy link
Contributor

i cant seem to get cibuildwheel to work on github ci 🤔

Error: cibuildwheel: 
Build failed because a pure Python wheel was generated.

If you intend to build a pure-Python wheel, you don't need
cibuildwheel - use `pip wheel .`, `pipx run build --wheel`, `uv
build --wheel`, etc. instead. You only need cibuildwheel if you
have compiled (not Python) code in your wheels making them depend
on the platform.

If you expected a platform wheel, check your project configuration,
or run cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.

@kevinjqliu kevinjqliu mentioned this pull request Oct 28, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kevinjqliu kevinjqliu kevinjqliu left review comments

@Fokko Fokko Fokko approved these changes

+1 more reviewer

@rambleraptor rambleraptor rambleraptor left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Switch from Poetry to uv for dependency management

4 participants

@geruh @kevinjqliu @Fokko @rambleraptor